Current natural language processing (NLP) tools can detect errors in text documents reflecting incorrect spelling, syntax, and grammar. These types of errors, however, do not relate to the underlying meaning of the subject matter of the text, i.e., they are not semantic errors.
These limitations have motivated the development of more sophisticated tools for analyzing natural language documents. One important application of such tools is automatic grading systems for summaries and essays in education. Most existing automated grading systems for student summaries are based on statistical models, such as latent semantic analysis (LSA) which detects statistical word similarity between a teacher's model document and a student's submitted document. If words occur with similar frequencies in the two documents, then the documents are considered to be statistically similar, and the student submission is given a high grade by the system. More specifically, LSA treats each essay as a matrix of word frequencies and applies singular value decomposition (SVD) to the matrix to find an underlying semantic space. Each student essay is represented in that space as a set of vectors. A similarity measure is then computed based on the cosine similarity between the vectors of the student essay and vectors of a model text document. The cosine similarity is then transformed to a grade and assigned to the student essay.
Although LSA and other semantic similarity techniques has proven to be very useful, they cannot detect logical errors which reflect a student's misunderstanding of the proper relationships between the words. Consequently, a student's essay that is semantically similar to an instructor's model essay but uses the terms in a logically incorrect manner would be inappropriately accorded a high grade. In short, LSA assigns inaccurate grades to student submissions that incorrectly use the correct terminology. In addition, because LSA is a statistical approach that treats each document as a whole, it cannot provide feedback about specific sentences in the document.
As an alternative to statistical approaches, some grading systems can identify correctly used concepts using a semantic network which represents the correct relationships between concepts of the subject matter. For example, an existing commercial system called SAGrader™ automatically analyzes student essays using a semantic network and provides feedback including confirming the correct relationships between concepts and identifying missing content. However, it is hard to use semantic network to express complex logic relationships such as negation and disjointness, which are fundamental for detecting logic errors.